Red Hat Enterprise Linux 7 Implementation

System Logging and NTP

Module Topics

System Log Architecture
Reviewing syslog Files
Reviewing systemd Journal Entries
Preserving the systemd Journal
Maintaining Accurate Time

System Log Architecture

System Logging

Processes and operating system kernel record event logs
- Use for system auditing and troubleshooting
Logs are persistently stored in /var/log
Red Hat Enterprise Linux 7 includes logging system based on syslog protocol
Syslog messages are handled by two services:
- systemd-journald
- rsyslog

System Log Architecture

systemd-journald Daemon

Collects messages from:
- Kernel
- Boot process
- Daemon output and errors
- Syslog
Writes messages to event journal
- By default, journal does not persist between reboots
- Journal allows missed syslog messages and events to be collected in database
Also forwards syslog messages to rsyslog for further processing

System Log Architecture

rsyslog Service

Sorts syslog messages by facility (type) and priority
Writes messages to persistent files in /var/log
/var/log holds system- and service-specific log files maintained by rsyslog

Log File

Purpose

/var/log/messages

syslog messages except those related to authentication and email processing (which periodically run jobs), and messages related to debugging

/var/log/secure

Security and authentication messages and errors

/var/log/maillog

Mail server messages

/var/log/cron

Messages of periodically executed tasks

/var/log/boot.log

Messages related to system startup

References

systemd-journald.service(8), rsyslogd(8), and rsyslog.conf(5) man pages
Red Hat Enterprise Linux System Administrator’s Guide

Reviewing `syslog` Files

syslog Files

Each syslog message is categorized by:
- Facility: Type of message
- Priority: Severity of message
Eight priorities are standardized and ranked as follows:

Code

Priority

Severity

emerg

System is unusable

alert

Action must be taken immediately

crit

Critical condition

err

Non-critical error condition

warning

Warning condition

notice

Normal but significant event

info

Informational event

debug

Debugging-level message

Reference

rsyslog.conf(5) man page

Reviewing `syslog` Files

rsyslog Files

rsyslogd uses facility and priority to determine how to handle message
Message handling is configured by:
- /etc/rsyslog.conf
- *.conf
Configuration files stored in /etc/rsyslog.d
To change rsyslogd so it is not overwritten by updates to rsyslog, use customized *.conf files in /etc/rsyslog.d

Reviewing `syslog` Files

/etc/rsyslog.conf File

Directives in #### RULES #### section define where to save log messages
Left side of each line indicates facility and severity of log message that directive matches
Can use * wild card in facility and severity fields
Right side of each line indicates what file to save log message in
Log messages normally saved in /var/log
Message handled by rsyslog can appear in multiple log files
- To prevent that, set severity to none
- Does not add any messages directed to this facility to specified log file
Can print syslog messages to terminals of all logged-in users.
- Default rsyslog.conf does this for all messages with emerg priority

Transcript:

The #### RULES #### section of /etc/rsyslog.conf contains directives that define where log messages are saved. The left side of each line indicates the facility and severity of the log message the directive matches. The rsyslog.conf file can contain the character * as a wild card in the facility and severity field, where it either stands for all facilities or all severities. The right side of each line indicates what file to save the log message in. Log messages are normally saved in files in the /var/log directory.

Log files are maintained by the rsyslog service, and the /var/log directory contains a variety of log files specific to certain services. For example, the Apache Web Server or Samba write their own log files into a corresponding subdirectory of the /var/log directory.

A message handled by rsyslog can appear in multiple different log files. To prevent that, the severity field can be set to none, which means that none of the messages directed to this facility are added to the specified log file.

Instead of logging syslog messages to a file, they can be printed to the terminals of all logged-in users. In the default rsyslog.conf file, this is done for all messages that have "emerg" priority.

Reviewing `syslog` Files

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :musrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

References

rsyslog.conf(5) man page
/usr/share/doc/rsyslog-*/manual.html in rsyslog-doc, available from Red Hat Enterprise Linux 7 software channel

Reviewing `syslog` Files

Log File Rotation

To prevent filling up file system, logs are rotated by logrotate
Rotated log files are renamed with date extension
- Example: /var/log/messages, rotated October 30, 2014, became /var/log/messages-20141030
When log file is rotated:
- New log file is created
- Service that writes to it is notified
After defined number of rotations, old log file is discarded to free disk space
A cron job runs logrotate daily to see if logs need rotation
Most log files rotate weekly
- Some rotate faster or slower
- Some rotate based on size

Transcript:

Logs are "rotated" by the logrotate utility to keep them from filling up the file system containing /var/log/. When a log file is rotated, it is renamed with an extension indicating the date on which it was rotated: the old /var/log/messages file may become /var/log/messages-20141030 if it is rotated on October 30, 2014. Once the old log file is rotated, a new log file is created and the service that writes to it is notified.

After a certain number of rotations, typically after four weeks, the old log file is discarded to free disk space. A cron job runs the logrotate program daily to see if any logs need to be rotated. Most log files are rotated weekly, but logrotate rotates some faster, or slower, or when they reach a certain size.

Configuration of logrotate is not covered in this course. For more information, see the logrotate(8) man page.

Reviewing `syslog` Files

Analyzing a syslog Entry

rsyslog logs put oldest message on top and newest message at end

Entries are recorded in standard format

Example: Anatomy of /var/log/secure

<1>Feb 11 20:1:48 <2>localhost <3>sshd[1433]: <4>Failed password for student from 192.168.0.1 port 59344 ssh2

Reviewing `syslog` Files

Monitoring Log Files with tail

Monitoring log files can help reproduce problems and issues
To output last 10 lines of specified file, use tail -f /path/to/file
- Command continues to output new lines when written

To monitor for failed login attempts on one terminal, use ssh as root while user logs in

Example using server1.example.com:

[root@server1~]$ tail -f /var/log/secure
...
Feb 10 09:1:13 localhost sshd[2712]: Accepted password for root from 192.168.0.254 port 56801 ssh2
Feb 10 09:1:13 localhost sshd[2712]: pam_unix(sshd:session): session opened for user root by (uid=0)

Reviewing `syslog` Files

Sending a syslog Message with logger

To send messages to rsyslog service, use logger
By default, logger sends message to facility user with user.notice severity
- To specify otherwise, use -p
logger useful for testing rsyslog configuration changes

To send message to rsyslogd that is recorded in /var/log/boot.log:

[root;@server1 ~]$ logger -p local7.notice "Log entry created on server1.example.com.

References

logger(1), tail(1), rsyslog.conf(5), and logrotate(8) man pages
/usr/share/doc/rsyslog-*/manual.html provided by rsyslog-doc package
Red Hat Enterprise Linux System Administrator’s Guide

Reviewing `systemd` Journal Entries

Finding Events with journalctl

systemd journal stores logging data in structured, indexed binary file
- Includes extra information about log event
- Example: For syslog events, can include facility and priority of original message
systemd journal stored in /run/log
- Contents are cleared after reboot
- Can change setting

Reviewing `systemd` Journal Entries

To see complete system journal, starting with oldest log entry, use journalctl as root

[root@server1~]# journalctl
Feb 13 10:1:01 server1 run-parts(/etc/cron.hourly)[8678]: starting 0yum-hourly.cron
Feb 13 10:1:01 server1 run-parts(/etc/cron.hourly)[8682]: finished 0yum-hourly.cron
Feb 13 10:0:01 server1 systemd[1]: Starting Session 725 of user root.
Feb 13 10:0:01 server1 systemd[1]: Started Session 725 of user root.
Feb 13 10:0:01 server1 CROND[8687]: (root) CMD (/usr/lib64/sa/sa1 1 1)

Messages with priority notice or warning appear in bold
Messages with priority error and higher appear in red

Reviewing `systemd` Journal Entries

Using journalctl to Filter Log Entries

Limit journal searches to show only relevant output
By default, journalctl -n shows last 10 log entries
- Can specify number of entries to display
- Example: Display last 5 log entries
  [root@server1~]# journalctl -n 5

Reviewing `systemd` Journal Entries

When troubleshooting, filter journal output by priority
journalctl -p takes either name or number of priority level
- Shows entries for that level and higher
Priority levels: debug, info, notice, warning, err, crit, alert, and emerg
To filter journalctl output to messages with priority err and above:
```
[root@server1~]# journalctl -p err
```
To output last 10 journal entries, use journalctl -f
- Continues to output new entries when written
  [root@server1~]# journalctl -f

Reviewing `systemd` Journal Entries

Searching by Time Range

To limit output to specific time range, use journalctl with --since and --until
- Both options take YYYY-MM-DD hh:m:ss data format
- If date is omitted, defaults to today
- If time is omitted, defaults to whole day starting at 00:0:00
- Can use yesterday, today, and tomorrow with both options
To output all journal entries recorded today:
```
[root@server1~]# journalctl --since today
```
To output journal entries from 10th February, 2014, 20:0:00 to 13th February, 2014, 12:0:00:
```
[root@server1~]# journalctl --since "2014-02-10 20:0:00" --until "2014-02-13 12:00:00"
```

Reviewing `systemd` Journal Entries

Using Verbose Output

Turn on verbose output to view extra fields for log entries

To reduce output of complex searches, use extra fields to filter query output

[root@server1~]# journalctl -o verbose
Thu 2014-02-13 02:6:00.409345 EST [s=0b47abbf995149c191a8e539e18c3f9c;
i=d28;b=1ea26e84667848af9a4a2904a76ff9a5;m=4d6878ff5a;t=4f244525daa67;
x=880bc65783036719]
    PRIORITY=6
    _UID=0
    _GID=0
    _BOOT_ID=1ea26e84667848af9a4a2904a76ff9a5
    _MACHINE_ID=4513ad59a3b442ffa4b7ea88343fa55f
    _CAP_EFFECTIVE=0000001fffffffff
    _TRANSPORT=syslog
    SYSLOG_FACILITY=10
    SYSLOG_IDENTIFIER=sshd
    _COMM=sshd
    _EXE=/usr/sbin/sshd
    _SYSTEMD_CGROUP=/system.slice/sshd.service
    _SYSTEMD_UNIT=sshd.service
    _SELINUX_CONTEXT=system_u:ystem_r:sshd_t:s0-s0:c0.c1023
    _HOSTNAME=server1.example.com
    _CMDLINE=sshd:root [priv]
    SYSLOG_PID=6833
    _PID=6833
    MESSAGE=Failed password for root from 192.168.0.1 port 59371 ssh2
    _SOURCE_REALTIME_TIMESTAMP=1392275160409345

Reviewing `systemd` Journal Entries

Query Options

Useful when searching for lines relevant to particular process or event

Option

Description

_COMM

Name of the command

_EXE

Path to executable for process

_PID

Process PID

_UID

UID of user running process

_SYSTEMD_UNIT

systemd unit that started process

Can combine options
- Example: Show all journal entries related to processes started by systemd unit file sshd.service that also have PID 1182:
  [root@server1~]# journalctl _SYSTEMD_UNIT=sshd.service _PID=1182

References

journalctl(1) and systemd.journal-fields(7) man pages
Red Hat Enterprise Linux System Administrator’s Guide

Preserving the `systemd` Journal

Default systemd Journal

By default, systemd journal is kept in /run/log/journal
- Cleared when system reboots
- For most installations, journal that starts with last boot is sufficient
If /var/log/journal exists, systemd logs to that directory instead
- Advantage: Historic data is available immediately at boot
Even for persistent journal, not all data is kept forever
- Built-in log rotation mechanism triggers monthly
- Journal cannot exceed 10% of file system
- File system must have 15% or more free

Preserving the `systemd` Journal

Changing systemd Journal Limits

Tune systemd size limits in /etc/systemd/journald.conf

Current journal limits are logged when systemd-journald starts

To see top two lines of journalctl output:

[root@server1~]# journalctl | head -2
-- Logs begin at Wed 2014-03-05 15:3:37 CST, end at Thu 2014-03-06 21:57:54 CST. --
Mar 05 15:3:37 server1.example.com systemd-journal[94]: Runtime journal is using 8.0M (max 277.8M, leaving 416.7M of free 2.7G, current limit 277.8M).

To make systemd journal persistent, create /var/log/journal as root
```
[root@server1~]# mkdir /var/log/journal
```

Preserving the `systemd` Journal

Set /var/log/journal to be:

Owned by root and group systemd-journal

Have permissions 2755

[root@server1~]# chown root:ystemd-journal /var/log/journal
[root@server1~]# chmod 2755 /var/log/journal

Reboot system or send special signal USR1 as root to systemd-journald
```
[root@server1~]# killall -USR1 systemd-journald
```
- systemd journal now persistent across reboots
- journalctl -b can reduce output by showing only log messages since last boot
  [root@server1~]# journalctl -b

Preserving the `systemd` Journal

Debugging a System Crash

When debugging with persistent journal, usually limit query to reboot before crash
To indicate number of prior system boots to output, use -b with negative number
- Example: journalctl -b -1 limits output to previous boot

References

mkdir(1), systemd-journald(1), and killall(1) man pages
Red Hat Enterprise Linux System Administrator’s Guide

Maintaining Accurate Time

Setting Local Clocks and Time Zone

Synchronized system time is crucial for log file analysis across multiple systems
NTP: Standard way to provide and obtain correct time on Internet
- Can get accurate time information from public NTP services such as NTP Pool Project
Can use high-quality hardware clock to serve accurate time to local clients

Maintaining Accurate Time

timedatectl Command

To view time information, including current time, time zone, and NTP synchronization settings, use timedatectl

[student@server1 ~]$ timedatectl
      Local time:Thu 2014-02-13 02:16:15 EST
  Universal time:Thu 2014-02-13 07:16:15 UTC
        RTC time:Thu 2014-02-13 07:16:15
        Timezone:America/New_York (EST, -0500)
     NTP enabled:yes
NTP synchronized:no
 RTC in local TZ:no
      DST active:no
 Last DST change:DST ended at
                  Sun 2013-11-03 01:9:59 EDT
                  Sun 2013-11-03 01:0:00 EST
 Next DST change:DST begins (the clock jumps one hour forward) at
                  Sun 2014-03-09 01:9:59 EST
                  Sun 2014-03-09 03:0:00 EDT

Maintaining Accurate Time

Time Zone Database

To list database with known time zones, use timedatectl list-timezones

[student@server1 ~]$ timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
...

Time zone names are based on public database maintained by IANA
Time zones are named by continent or ocean, then largest city
- Example: Most of US Mountain time zone is "America/Denver"

Maintaining Accurate Time

Selecting Time Zone

Selecting correct time zone may not be intuitive
- Localities within time zone may have different rules for daylight saving time
- Example: Much of Arizona (US Mountain time) does not use daylight saving time and is in time zone "America/Phoenix"
To identify correct time zone, use tzselect
- Interactively prompts with questions about system location and outputs name of correct time zone
- Does not make any changes to system time zone setting

Maintaining Accurate Time

Changing System Time Zone

To change system time zone, use timedatectl set-timezone as user root

[root@server1~]# timedatectl set-timezone America/Phoenix
[root@server1~]# timedatectl
      Local time:Thu 2014-02-13 00:23:54 MST
  Universal time:Thu 2014-02-13 07:23:54 UTC
        RTC time:Thu 2014-02-13 07:23:53
        Timezone:America/Phoenix (MST, -0700)
     NTP enabled:yes
NTP synchronized:no
 RTC in local TZ:no
      DST active:n/a

Maintaining Accurate Time

Changing System Time and Date

To change current time and date, use timedatectl set-time
- Format YYYY-MM-DD hh:m:ss
- Can omit either date or time

Example: Change time to 09:0:00

[root@server1~]$ timedatectl set-time 9:0:00
[root@server1~]$ timedatectl
      Local time:Thu 2014-02-13 09:00:27 MST
  Universal time:Thu 2014-02-13 16:00:27 UTC
        RTC time:Thu 2014-02-13 16:00:28
        Timezone:America/Phoenix (MST, -0700)
     NTP enabled:yes
NTP synchronized:no
 RTC in local TZ:no
      DST active:n/a

Maintaining Accurate Time

Enabling NTP Synchronization

Use set-ntp to enable or disable automatic NTP synchronization
- Requires true or false argument

To turn on NTP synchronization:

[student@desktop1 ~]$ timedatectl set-ntp true

Maintaining Accurate Time

Configuring and Monitoring chronyd

chronyd synchronizes local hardware clock (RTC) to NTP servers
If no network connectivity, syncs to calculated RTC clock drift in driftfile in /etc/chrony.conf
By default, chronyd uses NTP Pool Project servers
- Does not need additional configuration
- Can change NTP servers for machines on isolated network
Quality of NTP time source determined by stratum value
- Indicates number of hops machine is away from reference clock
  - stratum 0: Reference clock
  - stratum 1: Directly attached to NTP server
  - stratum 2: Machine synchronizing from NTP server

Transcript:

The chronyd service keeps the usually-inaccurate local hardware clock (RTC) on track by synchronizing it to the configured NTP servers, or if no network connectivity is available, to the calculated RTC clock drift which is recorded in the driftfile specified in the /etc/chrony.conf configuration file.

By default, chronyd uses servers from the NTP Pool Project for the time synchronization and does not need additional configuration. It may be useful to change the NTP servers when the machine in question is on an isolated network.

The quality of an NTP time source is determined by the stratum value reported by the time source. The stratum determines the number of hops the machine is away from a high-performance reference clock. The reference clock is a stratum 0 time source. An NTP server directly attached to it is a stratum 1, while a machine synchronizing time from the NTP server is a stratum 2 time source.

Maintaining Accurate Time

server and peer Time Sources

Two categories of time sources configured in /etc/chrony.conf:
- server: one stratum above local NTP server
- peer: same stratum as NTP server
Can specify more than one server and peer
- Specify one per line
- First argument of server line is IP address or DNS name of NTP server
- Next is series of server options
  - Recommended: Use iburst option
  - After service starts, quickly takes four measurements for accurate initial synchronization

Maintaining Accurate Time

Reconfiguring chronyd

Can configure chronyd server to synchronize with server other than default servers
Example: Reconfigure to use instructor.example.com
1. In /etc/chrony.conf, replace current server entries with the following:
  # Use public servers from the pool.ntp.org project. server instructor.example.com iburst
2. Restart chronyd:
  [root@server1~]# systemctl restart chronyd

Maintaining Accurate Time

chronyc Command

chronyc command acts as client to chronyd service
After setting up NTP, verify that NTP server was used to synchronize system clock

To verify clock synchronization, use chronyc sources

To view verbose output, use chronyc sources -v

[root@server1~]$ chronyc sources -v
210 Number of sources = 1

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||                                                /   xxxx = adjusted offset,
||         Log2(Polling interval) -.             |    yyyy = measured offset,
||                                  \            |    zzzz = estimated error.
||                                   |           |
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* instructor.example.com         8   6    17    23   -497ns[-7000ns] +/-  956us

References

timedatectl(1), tzselect(8), chronyd(8), chrony.conf(5), and chronyc(1) man pages
Red Hat Enterprise Linux System Administrator’s Guide

Transcript:

The chronyc command acts as a client to the chronyd service. After setting up NTP synchronization, it is useful to verify the NTP server was used to synchronize the system clock. This can be achieved with the chronyc sources command or, for more verbose output with additional explanations about the output, chronyc sources -v:

The * character in the S (Source state) field indicates that the instructor.example.com server has been used as a time source and is the NTP server the machine is currently synchronized to.

Red Hat Enterprise Linux 6 and earlier use ntpd and ntpq to manage the NTP configuration. Further information may be found in the documentation for Red Hat Enterprise Linux 6.

Additional information about time synchronization may be available in the NTP Pool Project and Time Zone Database sections of the Red Hat Enterprise Linux System Administrator’s Guide for Red Hat Enterprise Linux 7, which can be found at docs.redhat.com.

Summary

System Log Architecture
Reviewing syslog Files
Reviewing systemd Journal Entries
Preserving the systemd Journal
Maintaining Accurate Time

Module Completion

Nice job!

Click the button below to complete this module of the course:

Red Hat Enterprise Linux 7 Implementation

System Logging and NTP

Module Topics

System Log Architecture

System Log Architecture

System Log Architecture

Reviewing syslog Files

Reviewing syslog Files

Reviewing syslog Files

Reviewing syslog Files

Reviewing syslog Files

Reviewing syslog Files

Reviewing syslog Files

Reviewing syslog Files

Reviewing systemd Journal Entries

Reviewing systemd Journal Entries

Reviewing systemd Journal Entries

Reviewing systemd Journal Entries

Reviewing systemd Journal Entries

Reviewing systemd Journal Entries

Reviewing systemd Journal Entries

Preserving the systemd Journal

Preserving the systemd Journal

Preserving the systemd Journal

Preserving the systemd Journal

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Maintaining Accurate Time

Summary

Module Completion

Reviewing `syslog` Files

Reviewing `syslog` Files

Reviewing `syslog` Files

Reviewing `syslog` Files

Reviewing `syslog` Files

Reviewing `syslog` Files

Reviewing `syslog` Files

Reviewing `syslog` Files

Reviewing `systemd` Journal Entries

Reviewing `systemd` Journal Entries

Reviewing `systemd` Journal Entries

Reviewing `systemd` Journal Entries

Reviewing `systemd` Journal Entries

Reviewing `systemd` Journal Entries

Reviewing `systemd` Journal Entries

Preserving the `systemd` Journal

Preserving the `systemd` Journal

Preserving the `systemd` Journal

Preserving the `systemd` Journal